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Abstract 

The COVID-19 pandemic led to an abrupt shift from in-person to virtual instruction in 
Spring 2020. Using a difference-in-differences framework that leverages within-course variation 
on whether students started their Spring 2020 courses in person or online, we estimate the impact 
of this shift on the academic performance of Virginia’s community college students. We find that 
the shift to virtual instruction resulted in a 6.7 percentage point decrease in course completion, 
driven by increases in both course withdrawal and failure. Faculty experience teaching a course 


online did not mitigate the negative effects of moving to virtual instruction. 
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Introduction 

The COVID-19 health crisis has led to one of the largest disruptions in the history of 
American higher education. Beginning in March 2020, tens of millions of college students who 
were enrolled in in-person courses abruptly had to shift to online learning. While some course 
instructors had experience teaching online, many faculty had to pivot into online teaching for the 
first time, often using relatively rudimentary technology (e.g. Zoom conferencing) to deliver 
instruction and engage students. 

Prior research demonstrates that health and economic disruptions can have substantial 
effects on students’ achievement and educational attainment. For instance, children born during 
the 1918 Influenza Pandemic received significantly fewer years of education as compared to 
their older siblings (Parman, 2015). More recently, students exposed to hotter learning 
environments through a combination of extreme temperatures and insufficient air conditioning in 
schools, perform worse academically (Park et al, 2020). The economic downturn from the Great 
Recession reduced the math and English Language Arts achievement of elementary and 
middle-aged students (Shores and Steinberg, 2017), and negative economic shocks arising from 
local area job losses during adolescence can have long-term educational and mental health 
implications, particularly among lower-income populations (Ananat et al, 2017). 

While several papers demonstrate that economic downturns affect college enrollments, 
institutional expenditures, and students’ choice of college majors (Barr and Turner, 2013; Barr 
and Turner, 2015; Betts and McFarland, 1995; Ersoy, 2020; Long, 2014), there has been less 
research on how downturns affect student success in higher education. Descriptively, students 


who initially enrolled in college during the Great Recession were less likely to graduate 


compared to earlier and more recent cohorts (Shapiro et al, 2016); however, this trend is likely 
driven in large part by compositional changes in the student population during this time. 

There are various reasons why the broader COVID-19 crisis and the ensuing abrupt shift 
to online learning may have led to worse negative outcomes for students. Based on a survey 
administered to college students in April 2020, Aucejo et al (2020) report that 13% of students 
report delaying graduation and 40% lost a job, job offer or internship due to the pandemic. 
Low-income students were much more likely to report having delayed graduation. Additionally, 
students may have been dealing directly with health challenges associated with COVID-19 
infection, or have had family members who became sick. Many students were among the tens of 
millions of Americans who lost their jobs during Spring 2020, or may have had family members 
lose employment; the stress of these job losses may have reduced the cognitive bandwidth and 
attention students could devote to class (Shah, Shafir, and Mullainathan, 2015). Increased child 
care responsibilities may have detracted from time adult students could invest in their college 
course work. 

Research to date on the efficacy of online versus in-person learning suggests that students 
tend to fare worse in online classes. Randomized studies which assigned students to either 
in-person or online learning have shown online learning to have negative impacts on course 
performance (Figlio, Rush and Yin 2013; Alpert, Couch and Harmon 2016). Additional studies 
which employ quasi-experimental methods find that online learning decreases course 
completion, final grade and enrollment persistence, and increases in course repetition (Xu and 


Jaggars 2011; Xu and Jaggars 2013; Hart, Friedman, and Hill 2016; Bettinger et al. 2017). 


At the same time, there are several reasons why the sudden shift to online learning may 
not have negatively affected student outcomes, or why the magnitude of this effect may have 
been not as profound as some might expect. For instance, the combined shift to online learning, 
remote work, and even job loss may have substantially increased the time available to students to 
invest in their courses. Forward-thinking students who anticipate the importance of additional 
training or education in a potentially transformed post-COVID economy may have renewed their 
efforts toward earning a credential. Students may have had additional access to asynchronous 
course materials to support their learning and exam preparation. Many colleges implemented 
emergency grading policies which could have reduced the effort required from students to pass 
their courses and make further progress towards their degree. Since the shift to online courses 
happened well into the Spring 2020 semester, negative impacts on student learning could also be 
mitigated by the opportunity to create relationships in-person prior to the switch and therefore 
had a higher degree of “social presence”, which is associated with better outcomes in online 
learning (Liu, Gomez, and Yen, 2009). Finally, most of the prior research on the efficacy of 
online learning has estimated local treatment effects or average treatment effects (from RCTs) 
that may not generalize beyond the experimental setting. These studies may not capture the 
relative impact of online versus in-person education in the COVID-19 context, which induced an 
effectively population-level shift to online education. 

Isolating the effect of the abrupt shift to online learning on student outcomes is 
challenging, given the lack of variation in the timing over which most colleges and universities 
shifted to online, and because of the parallel health, economic, and child care challenges that 


could have also affected students’ academic success. In this paper, we use a 


difference-in-differences estimation strategy, exploiting variation in whether students were 
enrolled in in-person or online classes at the start of the Spring 2020 semester, to produce the 
first causal estimates we know of on the impact of the sudden shift to online learning as a result 
of COVID-19. More specifically, we compare changes in course completion rates along two 
dimensions: (1) in-person versus online courses; (2) Spring 2020 versus recent comparison 
terms. We classify students enrolled in in-person courses at the start of the Spring 2020 as 
“treated”, i.e. they experienced the sudden shift to virtual instruction. We estimate our models 
with course fixed effects to respectively control for differences in student outcomes occurring 
across courses. 

We estimate the impact of abruptly shifting to online using student-level data from the 
Virginia Community College System, which enrolls approximately 250,000 students per year, 
on numerous dimensions is broadly representative of open access institutions across the country, 
and which had a broad slate of online course offerings prior to COVID-19. 

Our analyses yield several primary results. The move from in-person to virtual instruction 
resulted in a 6.7 percentage point decrease in course completion. This translates to a 8.5 percent 
decrease when compared to the pre-COVID course completion rate for in-person students of 79.4 
percent. This decrease in course completion was due to a relative increase in both course 
withdrawal (5.2 pp) and course failure (1.4 pp). We find very similar point estimates when we 
estimate models separately for instructors teaching both modalities versus only one modality, 
suggesting that faculty experience teaching a given course online does not mitigate the negative 
effects of students abruptly switching to online instruction. The negative impacts are largest for 


students with lower GPAs or no prior credit accumulation. 


Our paper makes several contributions. First, our analyses are the first we are aware of 
that estimate the causal effect of the abrupt shift to online instruction stemming from the 
COVID-19 crisis on students’ academic performance in the Spring 2020 semester. Our results 
make concrete the tradeoff in terms of student performance from abruptly shifting from 
in-person to online instruction that may arise if, during future semesters, higher education 
administrators need to cancel in-person classes because of COVID-19 resurgences or other 
future disruptions. The diminished performance in online courses that we estimate could be 
weighed alongside the costs of continuing in person, such as increased health risks to students, 
faculty, or staff, or the cost of investments in personal protective equipment. Our results 
moreover indicate that educators may need to invest additional resources to structure online 
learning environments that more closely resemble in-person instruction, such as having 
synchronous class meeting times, and building in additional opportunities for faculty and student 
engagement. 

Second, we make the novel contribution of demonstrating that declines in student 
performance stemming from disruptions are not solely a function of economic, health, or familial 
challenges that students experience outside of the classroom, or a function in overall shifts in 
institutional resources, and are also influenced by changes in course delivery and pedagogy 
induced by large-scale disruptions. Finally, our paper builds on a growing body of research 
demonstrating mixed and often negative impacts of online education compared with similar 
programs of study offered in-person, and extends prior research by demonstrating the negative 


effects of online education in the context of a much broader shift to virtual learning. 


Background 
Virginia Community College System 

The Virginia Community College System (VCCS) is comprised of 23 colleges across the 
Commonwealth, and in the 2018-19 academic year enrolled 228,135 students.’ The population 
that VCCS serves is similar to other community colleges and broad access institutions 
nationwide. For example, the demographic characteristics of VCCS students is similar to the 
broader community college landscape; at similar institutions, 49% of students are White or Asian 
and 37% are underrepresented minorities (URM). VCCS serves a slightly higher percentage of 
White and Asian students (58%) with 33% URM. Thirty-three percent of students at similar 
institutions receive Pell grants, compared to 29% at VCCS. The graduation rate in 150% of 


expected time to completion is 29% at similar institutions and 32% at VCCS. 


VCCS online course offerings 

Online courses are a well established practice at VCCS, dating back to 1996. Online 
instruction can take different forms, from “interactive classroom video” which consists of 
synchronous two-way video sessions between instructors and students, to fully asynchronous 
administered through a learning management system -- with the majority of VCCS online 
courses being offered through the latter. 

In the 2008-2009 academic year, 38.5% of the student population was enrolled in online 
learning, either exclusively or coupled with in-person courses.’ By the 2018-2019 academic year, 


this increased to 55.9%, with 4,235 unique courses and 12,122 individual course sections offered 


' Source: https://www.vccs.edu/about/#statistics 
? Source: https://www.vccs.edu/about/#statistics 


online. VCCS offers a wide range of courses online. In the 2018-2019 academic year, 64% of 
courses offered online were 100-level or lower (introductory courses) and 36% of courses were 


200-level courses. 110 unique subjects were taught online in 2018-2019. 


Changes within VCCS due to COVID 

In response to the COVID-19 crisis and the Governor’s declaration of a state of 
emergency on March 12, 2020, VCCS courses which started the Spring 2020 semester in-person 
were moved to virtual instruction. The switch to virtual happened on March 18, 2020 and 
courses remained virtual through the end of the Spring semester on May 11, 2020. On March 24, 
the chancellor of VCCS announced the system would switch to a Pass/No Pass emergency 
grading system for Spring 2020. The emergency grading system consisted of four grading 
options: P+, indicating the course credit is transferable and counts towards VCCS degree 
requirements; P-, indicating the course credit is not transferable but still counts towards VCCS 
degree requirements; incomplete; and withdrawal. While the emergency grading system was the 
default, students had the option of opting-in to receiving a traditional letter grade (A-F). In 


practice, 71% of students opted-in to the traditional grading scale for at least one of their courses. 


Empirical Strategy 
Data 

Data for this study come from systemwide administrative records for students enrolled in 
credit-bearing coursework at a VCCS college, beginning in Fall 2000. For each term in which a 


particular student was enrolled, these records contain detailed academic information including 


the program of study the student was pursuing (e.g. an Associate of Arts & Sciences in Liberal 
Arts); which courses and course sections the students were enrolled in (e.g. ENG 111 taught by 
Instructor X, MWF 9-10am), the grades they earned, and any VCCS credentials awarded. The 
data also contain information about each course and course section, including the modality of 
instruction (online, in-person), an  instructor-specific identifier, and basic instructor 
characteristics (sex, gender, full-time versus adjunct status). We also observe basic demographic 
information about each student, as well as National Student Clearinghouse matches starting in 


2005. 


Analytic Samples 
Our analytic sample consists of student x course level observations from the five most 
recent Spring terms (2016 through 2020). When we refer to a course, we treat the same general 
course taught at different colleges as separate courses; for example, we treat ENG 111 at 
Piedmont Virginia Community College as a distinct course from ENG 111 taught at Northern 
Virginia Community College. We focus on Spring terms because the population of VCCS 
students varies meaningfully between Spring, Summer, and Fall terms due to differential student 
attrition. We exclude observations (in both Spring 2020 and the prior comparison terms) 

corresponding to:* 
e Dual enrollment students. We expect that the transition from in-person to virtual 


instruction may have been operationalized in a significantly different manner for dual 


3 We also exclude hybrid courses, which are a small share of VCCS course offerings and are defined as “the 
combination of face-to-face and electronic delivery where 50-99% of the course content is electronically delivered”. 
When we instead classify hybrid courses as online and include those observations in our sample, our results are 
similar. We also exclude observations representing a student auditing a course, although these are very rare. 


enrollment classes, as many of these courses are taught in high schools by high school 
faculty. In addition, the vast majority of dual enrollment courses are offered in-person. 

e Courses offered outside the full session. While the majority of VCCS courses are offered 
within the full session, which lasts 15 or 16 weeks and spans January through May (with 
exact start and end dates depending on the college), some courses are offered during 
shorter sessions. The shorter sessions during the first half Spring 2020 were largely or 
entirely unaffected by COVID because they ended during March 2020, while the shorter 
sessions during the second half of Spring 2020 were fully online, and some students may 
have decided not to attempt these courses due to COVID. 

e Developmental courses. The vast majority of developmental courses, which are not 
credit-bearing, are offered during the abbreviated sessions. Additionally, many VCCS 
colleges have made meaningful changes to their developmental course policies in recent 
years, resulting in significant decreases in the share of students required to take 


developmental courses. 


After these restrictions, our overall sample contains 1,100,087 student x course observations, 
corresponding to 295,515 unique students. 

We further restrict the sample to include courses that were taught both online and 
in-person during Spring 2020 and were taught both online and in-person during at least one of 
the pre-COVID comparison terms. Given that online course offerings changed within our sample 
(i.e. some courses were newly offered online during the 2019-20 academic year which had 


previously only been offered in-person), our results would be biased if student outcomes differed 


meaningfully in the newly offered courses compared to those that have been offered for longer 
periods. This restriction also removes all courses that could not move to virtual instruction 
during Spring 2020 (e.g. clinical courses as part of Nursing rotations); no students enrolled in 
Spring 2020 received credit for such courses. We refer to the analytic sample with this restriction 
imposed as the “within-course” sample.* Within Spring 2020, 37% of student x course 
observations and 46% of unique students from the overall sample are present in the 
within-course sample. 

In supplementary analyses, we further limit the sample to either: (1) courses taught by the 
same instructor, both online and in-person, during both Spring 2020 and at least one earlier term, 
which we refer to as the “within-instructor” model; and (2) courses taught by instructors who 
have only ever taught that course via one modality, online or in-person, during both Spring 2020 
and at least one earlier term, which we refer to as the “across-instructor” model. Both the 
within-instructor and across-instructor samples are subsets of the within-course sample, so that 
we are still focused on courses that are offered both online and in-person during Spring 2020 and 


at least one comparison term. 


Difference-in-differences models 
Our primary empirical specification is a difference-in-differences model with course 


fixed effects, represented in the following regression equation: 


Vecit = Bo +B, InPerson,.,, + B, COVID, + B,COVID, * InPerson,,,,+ 


seit 


4 We also attempted a student-level model where the main outcome of interest was the share of attempted credits the 
student completed within a particular term. However, this student-level model violated the parallel trends 
assumption of the difference-in-differences model. 


- BAX scr bi Bs Wi a B,Z; - CourseFE . > Esict (1) 


where y_.,, is the course outcome for student (s) in course (c) taught by instructor (1) in term (t). 


sict 
Our primary outcome of interest is course completion; we set this binary outcome to one if the 
student received a grade of A, B, C, D, P+, or P- and to zero if otherwise. We also estimate the 
model separately for the outcomes of whether the student withdrew from or failed the course. 

We are unable to use the outcome of grade points (e.g. A = 4.0), because there are no grade 
points assigned to the P+ or P- grades as part of VCCS’s emergency grading policy. /nP erson, 
is an indicator equal one for if the student was enrolled in an in-person section of the course, and 
zero for online; COV ID, 1s an indicator equal to one for Spring 2020 and zero for the 
comparison terms. X,.,1s a set of student-level covariates to control for basic demographics, 
program of study fixed effects, academic experiences at VCCS prior to term ¢ (number of credits 
accumulated prior, cumulative GPA, etc), prior academic experiences at non-VCCS colleges 
prior to term ¢, and enrollment count of the section of course c in which the student was enrolled. 
W ,, 1S a set of time-variant instructor-level covariates that includes tenure at VCCS (measured in 
number of terms as a VCCS instructor since Spring 2008, which is the first term during which 
we reliably observe the instructor-specific identifier), and full-time versus adjunct status; Z, is a 
set of time-invariant instructor-level covariates that includes sex and race. While we treat the 
data as a repeated cross section, students can appear in multiple terms in the data. Therefore, we 


cluster the standard errors at the student level. The difference-in-differences estimate (,) 


measures the impact of the move from in-person to virtual instruction on course completion. 


We also estimate a within-instructor model with instructor x course fixed effects, 
restricting the sample to instructors who taught course c both online and in-person during Spring 


2020 and at least one comparison term: 


Vecit = By +B, lnPerson,,, + B, COVID, +B,COV ID, * InPerson,.,,+ 


seit 


+ B,X, +B 5W,, + InstructorxCourseF E,, + €,,, (2) 


Finally, we estimate an across-instructor model, which uses the same equation (1) shown 
above, but restricts the sample to instructors who taught course c during Spring 2020 and at least 


one comparison term, but who only taught the course via one modality, in-person or online. 


Results 
Summary statistics 

In Table 1, we present summary statistics for select student-, course-, and instructor-level 
characteristics. We compare this information for the overall sample (students who meet the 
initial restrictions outlined above) and within-course sample to demonstrate how our main 
analytic sample differs from the wider population of VCCS students in full-session, 
college-level, credit-bearing courses. All the information shown in Table | is measured as of 


Spring 2020; the patterns are very similar when considering the four earlier comparison terms.° 


° We focus on Spring 2020 for two reasons: (1) students may appear across multiple terms within the sample; since 
some of the student-level characteristics are time dependent, we need to choose one term; (2) the samples are 
restricted to courses which were definitely offered in Spring 2020, but not necessarily in each of the comparison 
terms. 


Comparing the first two columns of Panel A, we see that students in the within-course 
sample are slightly younger, more female, more White and Black and less Hispanic and Asian 
than students in the overall sample. Within-course sample students have similar academic 
histories as students in the overall sample, with slightly lower cumulative GPAs and fewer 
accumulated credits, and slightly more likely to have previous experience taking online courses 
at VCCS. The most notable student-level differences are that within-course students are more 
likely to be pursuing a Liberal Arts or other transfer-oriented associate degree program, and less 
likely to be pursuing applied or vocational/technical programs of study. This pattern is indicative 
of differences across programs of study in availability of online programming. As mentioned 
above, the within-course sample includes 46% all unique students from the overall sample, 
meaning nearly half of all students in the overall sample enrolled in at least one course that was 
offered both online and in-person in Spring 2020 and during at least one comparison term. 

The last two columns of Panel A compare the characteristics of students in the 
within-course sample who were enrolled in in-person versus online courses. Online students are 
older, are more likely to be female or White, and have higher GPAs and more credits 
accumulated. Not surprisingly, online students are 50% more likely to have previously taken an 
online course at VCCS, and have attempted a higher share of previous credits online. Finally, 
online students are slightly more likely to be pursuing applied degree and certificate programs. 

Panel B of Table 1 compares the characteristics of the courses represented in the overall 
versus within-course sample. Out of 4,722 courses offered in Spring 2020 meeting our overall 
sample criteria, only 663 courses (14%) meet the within-course sample criteria, though as we 


note above the sample accounts for 47% of students in the overall sample. The within-course 


sample contains a larger share of 100-level courses (versus 200-level), and are courses with 
significantly higher levels of enrollment. The within-course sample also includes a larger share 
of courses of “general education” courses: Math, English, History, and Biology courses make up 
40% of the within-course sample, compared to 15% of the overall sample. Because each course 
in the within-course sample must be offered both in-person and online, the only differences in 
the last two columns in Panel B are for in-person versus online enrollment. We see that in the 
average course more students enroll in the in-person version, although in-person sections tend to 
be slightly smaller. 

Panel C compares the instructor-level characteristics across the two samples. Other than 
instructors in the within-course sample having a slightly longer tenure, the two samples are very 
similar in terms of sex, race/ethnicity, and full-time (versus adjunct) status. Over one-third of 
unique instructors in the overall sample (n=4,610) are represented in the within-course sample 
(n=1,689). The last two columns compare the characteristics of instructors in the within-course 
sample who taught in-person versus online courses; note that 381 instructors taught both 
in-person and online courses and are thus represented in both columns. Online instructors are 
more likely to be female, are less racially diverse, are more likely to be full-time, and have 
longer tenures, compared to in-person instructors. 

Overall, the summary statistics for the within-instructor and across-instructor samples 
look quite similar to the within-course samples, with two notable exceptions. First, compared to 
the within-course sample, instructors in the within-instructor sample are twice as likely to be 
full-time (81.5% versus 40.6%) and have longer tenures at VCCS (29 versus 23 terms). 


However, the across-instructor sample looks quite similar to the within-course sample. Second, 


the number of students, courses, and instructors represented in the within-instructor model is 
significantly lower: within Spring 2020, there are 12,343 unique students, 238 unique courses, 
and 232 unique instructors represented in the within-instructor model. The sample reduction is 
less stark for the across-instructor sample, which contains 26,913 unique students, 525 unique 


courses, and 1,099 unique instructors from Spring 2020. 


Grade distribution 

Figure 1 shows the distribution of grades for student x course observations in the 
within-course sample across two dimensions: (1) online versus in-person courses; and (2) Spring 
2020 (the COVID impacted term) versus Spring 2019 (the most recent pre-COVID comparison 
term).° The pre-COVID distribution of grades for online students is more concentrated at the 
tails, with a larger share of online students earning either As or Fs compared to in-person. 
Online students were more likely to withdraw from a course pre-COVID. Both of these patterns 
translate to a lower pre-COVID course completion for online versus in-person observations. 
There is a significant reduction in failing grades and a significant increase in withdrawals for 
both online and in-person students in Spring 2020. The decrease in failing grades is likely due to 
a combination of positive selection into the A-F scale, as well as more lenient grading practices 
by VCCS instructors. The grades P+ and P- are only populated during Spring 2020 as part of 
VCCS’s emergency grading policy. The higher share of P+/P- grades among in-person 
observations indicate that a slightly lower share of in-person opted out of the emergency grading 


policy and into the A-F scale. Overall, when comparing Spring 2019 to Spring 2020 the 


6 This plot looks very similar when using earlier Spring comparison terms and for the within-instructor and 
across-instructor samples. 


relatively larger reduction in failing grades and the relatively smaller increase in course 
withdrawals for online students compared to in-person students suggest that the shift from 
in-person to virtual instruction led to lower rates of course completion; we show this explicitly 


within our difference-in-differences regression framework below. 


Event studies 

The key identifying assumption for our difference-in-differences model is parallel trends 
in the pre-COVID outcomes for the in-person and online observations. In this context, the 
parallel trends assumption is that the differences in outcomes between online and in-person 
students were stable in all the pre-COVID periods and would have remained consistent in the 
Spring 2020 term were it not for the shock of COVID. We show that our approach satisfies the 
parallel trends assumption by presenting an event study, based on a slightly adjusted version of 
equation (1): 
Outcome 


= Yo+ ¥,Spring2016, + y,Spring2017, + y,Spring2018, + y,Spring2020, + y;JnPerson,., 


sict 


+ y,Spring2016, * InPerson,., + yzSpring2017, * InP erson,., + YgSpring2018, * InP erson,., + 


+ Y¥,Spring2020, * InPerson,, + V¥igo9Xiet ViyWoge TV oZs + CourseP E+ &,,, (3) 


Note that we exclude the terms Spring2019, and Spring2019,* InPerson,., from the 


regression model so that Spring 2019 serves as the reference term. Therefore, in order for the 


parallel trends assumption to be satisfied, the estimates of y,, y,, and y, should all be 
statistically indistinguishable from zero. Figure 2 shows this to be the case for our main outcome 
of course completion for the within-course model (Panel A), the within-instructor model (Panel 
B), and the across-instructor model (Panel C). We observe very similar pre-COVID differential 
trend estimates for our alternative outcomes of course failure or course withdrawal. 

A separate identifying assumption of the difference-in-difference approach is that there 
was no differential sorting of students due to the onset of “treatment”. However, given the 
sudden and unanticipated nature of the COVID crisis during March 2020, when the full session 
courses were nearing the mid-term mark and well past the date when students could unregister 
for courses without receiving a “W” grade, differential sorting would be very unlikely. Still, we 
confirm this by estimating equation (3) using student characteristics in place of the outcome 
variable. Overall, these results confirm that there was no differential sorting of students. We do 
see some differential trends -- specifically, a growing age gap between in-person and online 
students, and a differential trend in whether students were previously enrolled online -- but these 
differences are small in magnitude and most likely represent overall trends in which types of 
students were choosing to enroll online. We control for these and other student characteristics in 


the regression model. 


Impact estimates of the shift to online learning 
We present our main results for the within-course model in Table 2, Panel A. Column (1) 
shows an estimated 6.7 percentage point decrease in course completion due to the shift from 


in-person to online instruction. Relative to the course completion rate among in-person 


observations in the pre-COVID comparison terms of 79.4%, this point estimate translates to a 
8.5% decrease. Columns (2) and (3) show that this reduction in course completion is primarily 
driven by a large increase in course withdrawals (5.1 pp, 62% increase relative to pre-COVID 
mean), but also by a modest increase in course failure (1.4 pp / 11.6% increase). Particularly 
given that students had to opt-in to the traditional grading scale in order to receive an “F’”, this 
impact on course failure suggests that the shift to virtual instruction had a meaningful negative 
impact even on those students who were confident enough in their ability to navigate online 
coursework that they actively opted out of the emergency grading policy. 

Panels B and C of Table 2 present the same results for the within-instructor and 
across-instructor models, respectively. Despite the differences in these two samples in 
instructors’ experience teaching the same course across modalities, we find very similar impact 
estimates. We even find a slightly larger impact when the sample is restricted to instructors who 
taught both online and in-person (7.1 pp / 8.8% decrease) compared to those who taught only one 
modality (6.3 pp / 7.9% decrease). The consistency of the impact estimates across our three 
models suggest that instructor familiarity with online teaching was not able to mitigate the 
negative impact for in-person students. Instead, the negative impacts on student outcome appear 
to be driven by struggles students had shifting to the online learning environment. 

We also estimate a number of alternative specifications. First, we limit the sample to 
students who were either enrolled fully online or fully in-person. We find very similar results 
(7.2 pp / 9.3% decrease) which are not statistically distinguishable from the main results in Table 
2, suggesting that in-person students still struggled with the transition to online even if they had 


concurrent experience with online coursework. Second, when we classify hybrid courses as 


online and include the corresponding observations in the sample, we find a smaller impact 
estimate (4.5 pp / 5.6% decrease; statistically distinguishable from the main results); this is 
intuitive, as hybrid courses also experienced some disruption from the shift to all virtual 
instruction, although not at the same level as the fully in-person courses. Third, when we include 
Fall terms in the pre-COVID comparison group, and have the pre-COVID comparison window 
start at Spring 2018, we again find a smaller impact estimate (4.9 pp / 6.1% decrease; statistically 
distinguishable from the main results). We hypothesize that this difference is driven by 
variability in the student population between Fall and Spring terms. Due to the traditional 
academic calendar year and relatively low rates of Fall-to-Spring retention at community 
colleges, students enrolled in Spring terms are on average higher performing and have higher 
baseline rates of course completion. 

Finally, we test for differential impacts across student subgroups according to prior 
academic history and basic demographic characteristics. Table 3 shows the impact estimates on 
course completion for the within-course model, with each column showing the results from a 
separate regression with the sample limited to students in the particular subgroup listed in the 
column heading. We observe the largest impacts for students with a baseline GPA in the bottom 
third (9.8 pp / 15.4%), compared to students with GPAs in the top third (3.3 pp / 3.6%). 
Similarly, we observe significantly larger impacts for students with no credit accumulation (8.8 
pp / 12.6%) compared to students who had previously earned at least 30 credits (5.4 pp / 6.2%). 
These first two comparisons show that higher performing and more experienced students were 
less impacted by the switch to virtual instruction, compared to lower performing and less 


experienced students. This result is in line with prior research that found random assignment to a 


hybrid course with an online component led to worse outcomes for lower-performing students 
but had no negative impact among higher-performing students (Joyce et al, 2015). One 
explanation is that higher performing students typically have better self-regulatory behaviors, 
which are thought to be particularly important for success in an online learning environment (see 
Li et al, 2020 for a thorough review). We also observe more negative impacts for male students, 
though do not find differential effects by age or race/ethnicity.. These subgroup patterns are 


similar for the within-instructor and across-instructor models (results not shown). 


Discussion 

Using a well-identified estimation strategy, we demonstrate that the abrupt shift to online 
learning as a result of the COVID-19 crisis led to a meaningful decrease in course completion 
among community college students in Virginia. Our results contribute to the growing strand of 
literature on online learning in higher education, and show that students struggled with the shift 
to virtual instruction despite any increased flexibility that accompanied the shift. This negative 
effect was particularly pronounced for lower-performing and less experienced students. The 
subgroup-specific patterns suggest that facing a similar situation in the future, institutions could 
target outreach efforts to students who are most likely to struggle with virtual learning. 

The consistency of the impact estimates across our three models suggest that instructor 
familiarity with online teaching was not able to mitigate the negative impact for in-person 
students. Instead, the impacts appear to be driven by student struggles with online learning. 
Faced with a similar need to abruptly shift students to online in the future, our results suggest 


that it would be important to target support services toward students, particularly those who do 


not have experience with online learning. In addition, it may be beneficial to provide guidance to 
instructors on how to create virtual environments that more closely resemble in-person settings 
instead of defaulting to a “typical” online course structure. 

One caveat is that VCCS implemented an emergency grading policy during Spring 2020 
designed to minimize the negative impact of COVID on student grades; instructors may have 
been more lenient with their grading. As such, we view these estimates as a lower-bound of the 
negative impact of the shift to virtual instruction. 

These results only represent the short-term impact of the COVID crisis on student 
outcomes. Because our empirical strategy identifies the impact of the shift to online learning 
based on within-course variation, we have limited ability to estimate the longer-term impacts of 
this shift. Yet as prior research both on the impact of health and economic disruptions and on 
online learning has demonstrated, the near-term negative effects we estimate may translate into 
longer-term reductions in students’ educational attainment. The concrete decline in academic 
performance we estimate could contribute to higher education administrators’ assessment of the 
costs of future shifts to online learning (e.g. in the face of another COVID-19 resurgence) 


alongside the costs of continuing in person. 
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Table 1: Summary statistics of students, courses, and instructors in overall and within-course analytic sample, 
Spring 2020 


Panel A: Student-level characteristics 


Within-Course 


Overall All In-person Online 
Demographic characteristics 
Age 24.83 24.06 22.97 25.19 
Female 56.8% 59.6% 55.4% 66.8% 
White 52.4% 57.7% 56.0% 61.3% 
Black 18.5% 21.4% 21.4% 20.4% 
Hispanic 14.3% 9.6% 10.6% 8.1% 
Asian 7.5% 3.7% 4.1% 3.1% 
Other Race 7.2% 7.6% 7.9% 71% 
Academic history 
Prior cumulative GPA 2.82 2.77 2.74 2.85 
Prior accumulated credits 29.19 25.74 23.68 29.01 
Previously enrolled at VCCS? 92.7% 91.4% 91.2% 92.4% 
Previous online enrollment at VCCS? 56.5% 60.8% 51.4% 78.4% 
Share of previously attempted credits online 19.0% 21.9% 13.7% 34.4% 
Broad program of study category 
Liberal Arts 38.9% 49.5% 51.6% 49.4% 
Health Sciences 11.8% 10.3% 9.5% 11.2% 
Applied Sciences 3.4% 1.3% 1.3% 1.1% 
Vocational / Technical 45.9% 38.8% 37.6% 38.3% 
Degree level pursuing 
Transfer-oriented associate 66.6% 75.1% 78.7% 72.1% 
Applied associate 24.2% 17.3% 14.4% 20.0% 
Certificate 2.6% 1.9% 1.5% 2.3% 
Career Studies Certificate (short-term) 5.4% 4.4% 4.2% 4.3% 
Other 1.1% 1.2% 1.2% 1.3% 
N 84648 39690 27396 18193 


Panel B: Course-level characteristics 


Within-Course 


Overall All In-person Online 

100-level 53.2% 62.1% 62.1% 62.1% 

Course enrollment 982 2724 1629 1022 

Section (class) enrollment, overall 15 22 20 24 
Course Subject 

Math 5.3% 13.4% 13.4% 13.4% 

English 4.1% 9.8% 9.8% 9.8% 

History 2.5% 8.3% 8.3% 8.3% 

Biology 3.0% 8.1% 8.1% 8.1% 

N 4722 663 663 663 

Panel C: Instructor-level characteristics 
Within-Course 

Overall All In-person Online 
Female 53.0% 54.8% 51.4% 60.9% 
White 78.6% 80.2% 719.3% 84.9% 

Black 12.6% 13.5% 14.0% 9.4% 

Hispanic 2.5% 2.2% 2.3% 2.3% 

Asian 5.5% 3.3% 3.4% 2.3% 

Other Race 0.8% 0.8% 1.0% 0.8% 

Tenure (terms) 20 23 22 26 
Full-time 40.3% 40.6% 42.4% 54.9% 

N 4610 1689 1336 734 


Notes: The Overall Sample excludes all observations corresponding to dual enrollment students, developmental courses, audited courses, and 
courses offered outside of the full-session. The Within-Course sample includes all observations corresponding to courses that are offered both 
online and in-person during Spring 2020, and also offered both online and in-person during at least one of the comparison terms. All information 
presented is for students enrolled in, courses offered during, and instructors teaching during the Spring 2020 term. The "Other Race" category 
includes students who identify as American Indian or Alaskan, Hawaiian or Pacific Islander, two or more races, or whose race is missing. If a 
student has no prior VCCS enrollment history, their value for previous online enrollment and share of previously attempted credits online are both 
set to zero. Courses in both the Overall and Within-Course samples are either 100-level or 200-level. Tenure is measures in number of terms the 
instructor taught at least one course between Spring 2008 and Spring 2020, inclusive, with a maximum of three terms within an academic year. 


Figure 1: Distribution of grades for observations in Within-Course in Spring 2019 and Spring 2020, by instructional modality 
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Figure 2: Event study plots 
Panel A: Within-course model 
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Panel B: Within-instructor model 
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Panel C: Across-instructor model 
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Notes: these plots show the coefficient estimates and 95% confidence intervals of the coefficients in the 
Term x In-Person indicators from equation (3) in the text, for the course completion outcome. 


Table 4: Difference-in-differences estimates of the impact of switch to virtual 


instruction 
Course 
Completion Withdrew Failed 
() (2) (3) 
Panel A: Within-Course Model 
DID Estimate -0.0672*** 0.05 14*** 0.0142*** 
(0.0038) (0.0036) (0.0018) 
Pre-COVID in-person outcome mean 0.794 0.0832 0.122 
R-squared 0.1617 0.0944 0.1023 
N 385,259 385,259 385,259 
Panel B: Within-Instructor Model 
DID Estimate -0.0711*** 0.0543*** 0.0144*** 
(0.0074) (0.0069) (0.0038) 
Pre-COVID in-person outcome mean 0.811 0.0719 0.117 
R-squared 0.2006 0.1394 0.1247 
N 52,234 52,234 52,234 
Panel C: Across-Instructor Model 
DID Estimate -0.0633*** 0.0490*** 0.0137*** 
(0.0051) (0.0047) (0.0027) 
Pre-COVID in-person outcome mean 0.803 0.0796 0.117 
R-squared 0.1685 0.1078 0.1113 
N 149,711 149,711 149,711 


Notes: within each panel, each column represents a separate regression using the model specified in equation 1 
(Panels A and C) or equation 2 (Panel B) in the text, with the outcome variable as noted in the column header. 
The received credit outcome is equal to one if the student earned a grade of A-D, P+, or P-, and is equal to zero 
if the student earned a grade of F, I, or W. *** p < 0.01; ** p < 0.05; p < 0.1 


Table 5: Subgroup-specific impacts on course completion from within-course model 


Panel A: Subgroups by prior academic history 


Tercile of prior cumulative GPA Prior credits accumulated 
Bottom Middle Third 0 1 to 14 15 to 29 30+ 
d) (2) (3) ea (5) (6) @) 
DID Estimate -0.0982*** = -0.0697*** — -0.0335*** -0.0882*** — -0.0766*** -0.0674%*** -0.0548*** 
(0.0087) (0.0067) (0.0048) (0.0144) (0.0077) (0.0080) (0.0055) 
Comparison mean 0.639 0.846 0.928 0.702 0.743 0.825 0.875 
R-squared 0.1240 0.0851 0.0705 0.1467 0.1829 0.1580 0.1503 
N 115,542 115,642 115,601 42,230 130,618 86,439 125,972 


Panel B: Subgroups by student demographics 


Age Gender Race/Ethnicity 
Underrep 
< 25 25+ Female Male Minority White/ Asian 
(8) (9) (0) (1) (2) (3) 
DID Estimate -0.0719*** — -0.0629*** -0.0545*** = -0.0787*** -0.0735*** -0.0628*** 
(0.0046) (0.0071) (0.0048) (0.0065) (0.0068) (0.0046) 
Comparison mean 0.787 0.820 0.814 0.770 0.748 0.823 
R-squared 0.1778 0.1483 0.1612 0.1670 0.1664 0.1522 
N 278,564 106,695 224,572 159,959 148,176 237,083 


Notes: each column within each panel represents a separate regression using the model specified in equation (1) in the text, restricted to the subgroup denoted 
by the column headers. This table is limited to results from the within-course model, with the outcome of "received credit". Note that students with no prior 


cumulative GPA are not included in the first three columns. The underrepresented minority category includes Black, Hispanic, and Other Race. *** p< 
0.01; ** p < 0.05; p < 0.1 


